Crowdsourcing of Formally-Specified Computations

Formally-specified computations are defined by logical statements (a.k.a. claims) interpreted in a computable model. Semantic Games (SGs) of claims provide novel answers to crowdsourcing challenges yet with several limitations. We provide a comprehensive analysis of the limitations of SGs when used for crowdsourcing and propose a new concept, called the Contradiction-Agreement Game (CAG), which builds on SGs and has desirable properties for successful crowdsourcing. The list of desirable properties includes a progress property (each CAG improves evaluation of scholars or adds to social welfare), cheating prevention through anonymity and cheating detection and fairness of the CAG payoff function. We describe a proof of concept implementation of a CAGbased crowdsourcing platform for formally-specified computational problems, called the Scientific Community Game (SCG). SCG uses a modular construct, called a lab, to group related claims and to solve labs incrementally through lab relations, which are themselves captured as labs. SCG has the flavor of a ”Wikipedia for Computations” where inventions are explicit and composable. Our proposed system has significant applications, in addition to crowdsourcing formally-specified computations. (1) The collaborative and self-evaluating nature of SGs provides a peer-based evaluation system for MOOCs on formal science topics. The peer-based evaluation is guaranteed to be fair, and it saves significant time for the teaching staff. (2) SGs provide a lower barrier of entry to making contributions to formal sciences through game play. It is a significant help to scientists to test claims using the crowd.